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Four applications of ordering theory analysis in the 
formative evaluation of an instructional program vhich is being 
developed for retarded children are described. Data from the 
pilot-testing of this program demonstrated two uses of ordering 
theory directly related to the formative evaluation of instruction: 
the testing of hypothesized relationships between concepts and the 
revision of the instruction based partly on the hierarchies which 
were determined among items, in addition, ordering theory analysis is 
used in the construction of both diagnostic^placement tests and 
criterion-ref ejrenced tests. Attention is given to the educational 
significance of these applications in reference to the products 
developed, as well as the dissemination of ordering theory as a tool 
for formative evaluators. (Author/RC) 
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The objective of this paper is to describe four applications of 
ordering theory analysis used in the formative evaluation of an 
instructional program, including: a) the testing of hypothesized 
relationships betv:een concepts, b) the revision of the instruction 
based partly on the. hierarchies which were determined among items, 
c) the development of a diagnostic placement test, and d) the 
development of an efficient criterion referenced test. 

Airasian and Bart (1970) introduced tree theory as an alternative 
measurement raodel. Subsequently it was renamed "ordering theory" so 
that it v;ould more closely mirror its Boolean alsebraic framev/ork. 
Ordering theory provides a m.eans of identifying the logical relation- 
ships among items, and "has as its primary intent either the testing 
of hypothesized hierarchies amon?; items or the determination of 
uiei.aiuuiefai atituug iLemb" (Bare Krus , 1971, p. 1). Bart and Krus 
conclude their paper by stating that a major practical use of ordering 
theory v;ould btr to analyze test data "so that rich perspective , 
directive, and diagnostic information can be provided for the teacher 
and oriier test users" -(p. 13). The intent of the present paper is 
to describe several such uses in the formative evaluation of the 
Money., Measurement and Time Program (Taylor, Thurlov:, & Turnure, 1973) 
which is being developed for educable mentally, retarded (EI^IR) children. 

Ordering theory analysis is being used in the evaluation of all 
instruction from this program, but only data from the Money Unit 
is presented in thii=; suramary. Tliis unit was pilot tested in three 
EiMR classes, with the mean chronological age of the 36 children in 
these classes being 3, and the mean 10 71. These children were given 
a 55 item pre-test, which included items to test each of the objectives 
of the unit. Portions of this test were given during instruction and 
the complete test was again given after all instruction in the unit 
was complete. Of the 55 items in the complete test, 24 relate to 
the identification , labeling, physical characteristics , and relative 
value of the individual coins (i.e., penny, nickel, dime, quarter, 
and half-dollar) and the generic "coins." The ordering theory analysis 
of these 24 items v;as supplemented by a traditional item-analysis, 
and these tivo analyses on the pre- and post-test provide the data 
source for the present paper. 
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As Bart and Krus have discust^ed, the problem o£ a statistical 
test for a prerequisite relationship between two items is unsolved. 
The alternative to statistical tests is the tolerance level technique, 
which is the primary procedure used to indicate v;hether a hypothesised 
prerequisite relationship between two items is accepted or rejected. 
However, it is necessary to set low tolerance levels (of errors) to 
insure that maximally rigorous results are identified and that weaker 
or even spurious relationships are not selected. In the present 
analyses, -03 was set as the tolerance level which meant that if 
more than one of the 36 subjects had a pattern which conflicted with 
a given prerequisite relationship, that relationship wais rejected. 

The raw data from the pre- test were subjected to ordering theory 
analysis to determine the hierarchical relationships among the items. 
These existing hierarchical relationships were then compared v;ith the 
hypothesized relationships iipon which the instruction was developed. 
The most important relationship hypothesized was that most, if not 
all, of the 20 items relating to the identification, labeling^ physical 
characteristics, and relative value of the individual coins were 
prerequisites to ordering the 5 coins, as to their relative value. 
Figure 1 presents an ordering diagram of the items which were found 
to be prerequisite to ordering the coins. As can be inferred from 
this figure, 10 of the 20 items on the individual coins were 'found 
to be prerequisites to the coin ordering item. Several other hypo- 
thesized relations were confirmed by the ordering theory analysis 
and no reversals of hypothesized relations were found. 

The second reason for the use of ordering theory analyses was 
to provide qualitative data to be used in the revision of the 
instruction. Since this revision was based, on nine other data 
sources, in addition to these hierarchical relationships, the exact 
input from the ordering theory analysis is difficult to describe in 
this summary. However, tv^o important revisions \^ere made largely on 
the basis of the hierarchical relationships which ^^7ere determined 
by ordering theory. 

The third reason for the use of ordering theory analyses was 
to assist in the development of diagnostic placement tests. The 
analyses of the pre- test data v/ere used to develop these placement 
tests. For example, the item at the top of the ordering-diagram 
in Figure 1 provides an excellent indication of v;hether the child 
has obtained most of the instructional objectives. Indeed, if a 
subject is correct on this item and three other selected tej^t items, 
the ordering theory analysis would predict correct performance on 
all 24 items with 97% certainty. Furthermore, if the subject missed 
one or more of these specified items, it would be possible to select 
items that were lower on these branches of the ordering hierarchy, 
anci in this way determine where the child should be placed in the 
instructional unit. The specific items for the diagnostic placement 
test are now being selected so as to be consistent witli the revised 
instruction and objectives of the Money Unit. 
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The final roasou for the use of orderinj^ theory was to provide 
qualitative data to aid in the development of criterion referenced 
tests. This use is an important one since these criterion referenced 
tests are to be administered by classroom teacliers , and excessively 
long, repetitious, or difficult to administer tests would at best 
be a burden to the classroom teacher. However, a short test,. the 
results of which are easy to interpret, would provide a help to the 
teacher, as well as substantially increasing the probability of 
obtaining meaningful, valid evaluation data. Since the effectiveness 
of ordering theory analysis increases as the number of test items 
is decreased, it is seen as being very appropriate for the development 
of efficient criterion-referenced tests (Airasian 5f Bart, 1971). 

The educational significance of the present application of 
ordering theory can be in part evaluated by examining the effective- 
ness and usability of the resulting instructional units and tests. 
However, the *most important contribution of this paper seems to us 
to be that it describes several applications of ordering theory v;hich 
could make this type of analysis a valuable tool in formative evalua- 
tion and test construction. It v:ould seem that the qualitative 
data base provided by the ordering theory analyses, and the strong 
conceptual appeal of the hierarchy approach to instruction (ef., 
Gagne, 1970) make further investigation of the educational applications 
of ordering theory a high prioi?ity for this and other R&D projects. 
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id. = identification 
la. = label 

ph. = physical charac- 
teristics 

r.v. = relative value 



P = Penny 

N = Nickel 

D = Dime 

Q = Quarter 

HP = Half-Dollar 

C = Coins 



